Performance Comparison: yajl-ruby vs JSON::ResumableParser

I conducted a performance test to evaluate the efficiency of JSON::ResumableParser compared to the current yajl-ruby implementation.

Test Scenario

• A client sends a continuous stream of ~1MB JSON messages via TCP at fixed 100ms intervals (approx. 10 MB/s).
• Monitored CPU and Memory (RSS) usage of the receiver during processing.

Results (Handling ~10 MB/s constant load)

• yajl-ruby

  • CPU Usage: ~70% (Average)
  • Memory (Max RSS): ~126 MB

• JSON::ResumableParser

  • CPU Usage: ~5% (Average)
  • Memory (Max RSS): ~127 MB

JSON::ResumableParser processes the same data stream vastly more efficiently than yajl-ruby. While memory consumption remains nearly identical, the new parser reduces CPU usage from ~70% to just ~5% under the same load. This massive reduction in CPU overhead indicates that ResumableParser provides significantly higher potential throughput and overall resource efficiency in stream processing environments.

Fluentd conf

<source>
  @type forward
  bind "0.0.0.0"
  port 24224
</source>

<match **>
  @type null
</match>

Client script

require 'socket'
require 'json'
require 'time'
require 'securerandom'

host = '127.0.0.1'
port = 24224

puts "Connecting to #{host}:#{port}..."
sock = TCPSocket.new(host, port)

loop do
  msg = [
    "test.memory",
    Time.now.to_i,
    {
      "id" => SecureRandom.uuid,
      "message" => "This is a memory test for JSON::ResumableParser",
      "dummy_data" => ('a'..'z').to_a.sample * 1024 * 1024
    }
  ]

  json_str = JSON.generate(msg)
  sock.write(json_str)
  sleep 0.1
end

sock.close